Text-based games are a popular testbed for language-based reinforcement learning (RL). In previous work, deep Q-learning is commonly used as the learning agent. Q-learning algorithms are challenging to apply to complex real-world domains due to, for example, their instability in training. Therefore, in this paper, we adapt the soft-actor-critic (SAC) algorithm to the text-based environment. To deal with sparse extrinsic rewards from the environment, we combine it with a potential-based reward shaping technique to provide more informative (dense) reward signals to the RL agent. We apply our method to play difficult text-based games. The SAC method achieves higher scores than the Q-learning methods on many games with only half the number of training steps. This shows that it is well-suited for text-based games. Moreover, we show that the reward shaping technique helps the agent to learn the policy faster and achieve higher scores. In particular, we consider a dynamically learned value function as a potential function for shaping the learner's original sparse reward signals.

Developed by Nick Walton, AI Dungeon 2 is based on an open-source text-generating algorithm created by OpenAI. The AI was trained by feeding it choose-your-own-adventure stories from the Choose Your Story website. Think of AI Dungeon 2 as a Zork-inspired text adventure game (TAG), but rather than limiting players to only going in specific directions, it responds to pretty much anything the player types. AI Dungeon 2 is a first of its kind. Players start by choosing the type of adventure they want to play, with genres including: mystery, apocalyptic, zombies, and fantasy.

Abstract--In 2016, 2017, and 2018 at the IEEE Conference on Computational Intelligence in Games, the authors of this paper ran a competition for agents that can play classic text-based adventure games. This competition fills a gap in existing game AI competitions that have typically focussed on traditional card/board games or modern video games with graphical interfaces. By providing a platform for evaluating agents in textbased adventures, the competition provides a novel benchmark for game AI with unique challenges for natural language understanding and generation. This paper summarises the three competitions ran in 2016, 2017, and 2018 (including details of open source implementations of both the competition framework and our competitors) and presents the results of an improved evaluation of these competitors across 20 games. I. INTRODUCTION Before the widespread availability of graphical displays, text adventures were one of the few game genres that owed their existence solely to computing. The first text adventure was Colossal Cave (also known simply as Adventure), written in 1976 by Will Crowther for the PDP-10 mainframe [1]. With the advent of home computing in the late 1970s, Colossal Cave and other games such as Zork were enjoyed by many. The majority of early text adventures used a narration-action loop that accepted simple commands of the general form VERB or VERB NOUN (e.g. In response to such commands, the programs provided a description of the immediate environment, e.g. 'You are in an open field on the west side of a white house with a boarded front door. There is a small mailbox here.'

Autonomous agents must often detect affordances: the set of behaviors enabled by a situation. Affordance detection is particularly helpful in domains with large action spaces, allowing the agent to prune its search space by avoiding futile behaviors. This paper presents a method for affordance extraction via word embeddings trained on a Wikipedia corpus. The resulting word vectors are treated as a common knowledge database which can be queried using linear algebra. We apply this method to a reinforcement learning agent in a text-only environment and show that affordance-based action selection improves performance most of the time. Our method increases the computational complexity of each learning step but significantly reduces the total number of steps needed. In addition, the agent's action selections begin to resemble those a human would choose.